Configuration of a vehicle based on collected user data

ABSTRACT

A vehicle is configured to perform at least one action (e.g., control of acceleration or navigation of the vehicle) based on analysis of data that is collected regarding a user of the vehicle. In one embodiment, the data is collected from various sources (e.g., computing devices associated with the user) prior to usage of the vehicle by the user. For example, data may be collected from an intelligent appliance located in a building or other fixed structure in which the user lives, or in which the vehicle is stored or charged. The data collected from the fixed structure may relate to activities performed by the user while inside the fixed structure and/or relate to data associated with electronic communications of the user.

RELATED APPLICATIONS

This application is related to U.S. Non-Provisional application Ser. No.15/848,630, filed Dec. 20, 2017, entitled “Control of Display Device forAutonomous Vehicle,” by Junichi Sato, the entire contents of whichapplication is incorporated by reference as if fully set forth herein.

FIELD OF THE TECHNOLOGY

At least some embodiments disclosed herein relate to configuration ofcomputing devices for vehicles in general, and more particularly, butnot limited to, configuring actions performed by a vehicle based onanalysis of data that is collected regarding a user of the vehicle(e.g., an autonomous or other vehicle).

BACKGROUND

A user of a vehicle can be a driver in the case of a manually-drivenvehicle. In other cases, such as for an autonomous vehicle, the user ofthe vehicle typically performs fewer control actions than a “driver” asregards the operation of the vehicle. For example, in some cases, theuser may simply select a destination to which the vehicle travels, butwithout performing any directional or other control of the immediatemovement of the vehicle on the roadway.

Recent developments in the technological area of autonomous drivingallow a computing system to operate, at least under some conditions,control elements of a vehicle without the assistance from a user of thevehicle. For example, sensors (e.g., cameras and radars) can beinstalled on a vehicle to detect the conditions of the surroundings ofthe vehicle on a roadway.

A computing system installed on the vehicle analyzes the sensor inputsto identify the conditions and generate control signals or commands forthe autonomous adjustments of the direction and/or speed of the vehicle,without any input from a human operator of the vehicle. Autonomousdriving and/or advanced driver assistance system (ADAS) typicallyinvolves an artificial neural network (ANN) for the identification ofevents and/or objects that are captured in sensor inputs.

In general, an artificial neural network (ANN) uses a network of neuronsto process inputs to the network and to generate outputs from thenetwork. Each neuron m in the network receives a set of inputs p_(k),where k=1, 2, . . . , n. In general, some of the inputs to a neuron maybe the outputs of certain neurons in the network; and some of the inputsto a neuron may be the inputs to the network as a whole. Theinput/output relations among the neurons in the network represent theneuron connectivity in the network.

Each neuron m has a bias b_(m), an activation function f_(m), and a setof synaptic weights w_(mk) for its inputs p_(k) respectively, where k=1,2, . . . , n. The activation function may be in the form of a stepfunction, a linear function, a log-sigmoid function, etc. Differentneurons in the network may have different activation functions.

Each neuron m generates a weighted sum s_(m) of its inputs and its bias,where s_(m)=b_(m)+w_(m1)×p₁+w_(m2)×p₂+ . . . +w_(mn)×p_(n). The outputa_(m) of the neuron m is the activation function of the weighted sum,where a_(m)=f_(m) (s_(m)).

The relations between the input(s) and the output(s) of an ANN ingeneral are defined by an ANN model that includes the data representingthe connectivity of the neurons in the network, as well as the biasb_(m), activation function f_(m), and synaptic weights w_(mk) of eachneuron m. Using a given ANN model, a computing device computes theoutput(s) of the network from a given set of inputs to the network.

For example, the inputs to an ANN network may be generated based oncamera inputs; and the outputs from the ANN network may be theidentification of an item, such as an event or an object.

For example, U.S. Pat. App. Pub. No. 2017/0293808, entitled“Vision-Based Rain Detection using Deep Learning”, discloses a method ofusing a camera installed on a vehicle to determine, via an ANN model,whether the vehicle is in rain or no rain weather.

For example, U.S. Pat. App. Pub. No. 2017/0242436, entitled “RoadConstruction Detection Systems and Methods”, discloses a method ofdetecting road construction using an ANN model.

For example, U.S. Pat. Nos. 9,672,734 and 9,245,188 discuss techniquesfor lane detection for human drivers and/or autonomous vehicle drivingsystems.

In general, an ANN may be trained using a supervised method where thesynaptic weights are adjusted to minimize or reduce the error betweenknown outputs resulted from respective inputs and computed outputsgenerated from applying the inputs to the ANN. Examples of supervisedlearning/training methods include reinforcement learning, and learningwith error correction.

Alternatively or in combination, an ANN may be trained using anunsupervised method where the exact outputs resulted from a given set ofinputs is not known a priori before the completion of the training. TheANN can be trained to classify an item into a plurality of categories,or data points into clusters.

Multiple training algorithms are typically employed for a sophisticatedmachine learning/training paradigm.

The disclosures of the above discussed patent documents are herebyincorporated herein by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings in which like referencesindicate similar elements.

FIG. 1 illustrates a system to configure at least one action of avehicle based on collected data for a user of the vehicle, according toone embodiment.

FIG. 2 shows an example of a vehicle configured using an ArtificialNeural Network (ANN) model trained using collected data for a user ofthe vehicle, according to one embodiment.

FIG. 3 shows a method to configure at least one action performed by avehicle based on collected data of a user, according to one embodiment.

FIG. 4 shows a method to control a vehicle including performing anaction based on at least one output from a machine learning model,according to one embodiment.

FIG. 5 shows an autonomous vehicle configured based on collected userdata and controlled using a computer model, according to one embodiment.

FIG. 6 shows a vehicle configured for a user based on collected data ofthe user, according to one embodiment.

FIG. 7 is a block diagram of an autonomous vehicle including one or morevarious components and/or subsystems, each of which can be updated invarious embodiments to configure the vehicle based on collected userdata, according to one embodiment.

FIG. 8 is a block diagram of a centralized autonomous vehicle operationssystem, according to various embodiments.

DETAILED DESCRIPTION

At least some embodiments disclosed herein configure a vehicle for auser to perform at least one action (e.g., control of acceleration ofthe vehicle). The vehicle is, for example, a manually-driven vehicle oran autonomous vehicle. The configuration is based on analysis of datathat is collected regarding the user of the vehicle.

In one embodiment, the data is collected from various sources prior tousage of the vehicle by the user. For example, data may be collectedfrom a building or other fixed structure in which the user lives, works,or otherwise is present. The data collected from the fixed structure mayrelate to activities performed by the user when inside the fixedstructure.

In other cases, the data collected from the fixed structure may relateto observations of the user. The collected data may include, forexample, audio data collected by a microphone located within the fixedstructure and/or image data collected by a camera located within thefixed structure. In other embodiments, alternatively and/oradditionally, other types of sensors associated with the fixed structuremay be used to collect data associated with the user.

In some embodiments, the vehicle communicates with a computing device ofthe fixed structure. The vehicle can both receive and send data to thefixed structure. For example, the vehicle can send data that initiates achange in the environment or other function of the fixed structure. Inone example, data obtained from the vehicle is used to configure heatingprovided in a house of the user.

In some embodiments, the collected data may include data collectedduring usage of the vehicle. In other embodiments, the collected datamay include data collected during usage of a different vehicle, forexample whether owned by the user or rented.

In various embodiments, the configuration of one or more actionsperformed by the vehicle may include, for example, actions related tooperation of the vehicle itself and/or operation of other systemcomponents mounted in the vehicle and/or otherwise attached to thevehicle. For example, the actions may include actions implemented viacontrols of an infotainment system, a window status, a seat position,and/or driving style of the vehicle. For example, the infotainmentsystem may be configured to provide access to media of the user in amanner related to volume, sequence, etc. as desired by the user based onanalysis of the collected data.

In another example, the driving style of the vehicle can be configuredto be, for example, aggressive or passive. An aggressive driving stylecan, for example, include faster acceleration and braking, and morefrequent passing of other vehicles on the road.

In some embodiments, the analysis of the collected data includesproviding the data as an input to a machine learning model. The vehicleis controlled by performing one or more actions that are based on anoutput from the machine learning model.

In one example, a machine learning model is trained and/or otherwiseused to configure a vehicle to a user (e.g., tailor actions and/or anenvironment of the vehicle interior). For example, the machine learningmodel may be based on pattern matching in which prior patterns of userbehavior or other user data is correlated with desired characteristicsor configuration(s) for operation of the vehicle. The machine learningmodel may further use driver inputs during usage of the vehicle tocontrol actions of the vehicle. For example, the driver may inputselections into a user interface located in the vehicle.

In some embodiments, the fixed structure is a house in which the userlives. The collected data includes data collected from one or moresensors located in the house and/or from a mobile device of the user(e.g., via a wireless interface of the mobile device). The collecteddata is analyzed and used to configure a vehicle that the user desiresto use. For example, the response and/or actions of the vehicle can beconfigured based on analysis of the collected data.

In one embodiment, the collected data is analyzed to determine a mood ofthe user. For example, if the user is in a positive mood, the vehiclecan be configured to play more energetic or louder music on anentertainment system of the vehicle. In contrast, if the user is in anegative mood, the vehicle can be configured to play peaceful music at asofter volume level.

Data can be collected from various sources associated with the user. Forexample, sensors of smart home appliances in the home of the user can beused to collect data. In addition, routine interactions of the user withappliances, a mobile device, and/or other devices can be used todetermine a mood of the user.

In some embodiments, voice recognition can be performed to analyze audiodata obtained from the user. For example, the audio data can becollected by a microphone of a mobile device of the user and/or amicrophone located in a fixed structure which the user is located. Inone embodiment, the voice recognition is performed to determine a moodof the user.

In one example, the collected data may include sensor data collected bythe vehicle during its real world services when the user is a driver ora passenger. In one embodiment, the collected data is transmitted fromthe vehicles to a centralized server, which performs machinelearning/training, using a supervised method and the collected user dataand/or other sensor data, to generate an updated ANN model that can besubsequently loaded into the vehicle to replace its previously-installedANN model. The model is used to configure the operation of the vehiclefor the user.

In some embodiments, the collected data is additionally and/oralternatively collected from a mobile device or other computing deviceof the user (e.g., text or instant messages sent or received by theuser). For example, data obtained from and/or regarding electroniccommunications associated with the user can be collected and analyzed.

In some embodiments, the data collected from a mobile device or othercomputing device of the user includes, for example, calendar and/or taskinformation stored on such device. In the case of data obtained fromand/or regarding electronic communications associated with the user, thedata can relate to a destination of the user. This destination can, forexample, be used to configure one or more actions performed by thevehicle that takes the user to the destination.

In one embodiment, data is collected from a wearable computing device.For example, data regarding heart rate or other characteristics of auser can be obtained using the wearable computing device. In othercases, biometric data regarding historical characteristics of the usercan be collected (e.g., prior heart rate at certain times or duringcertain activities).

In another embodiment, the collected data in part includes datacollected while the user is driving and/or interacting with the vehicle.For example, collected data may used to detect that a driver level ofawareness has fallen below safe levels, such as when the driver isdrowsy. In some examples, the vehicle can take over certain operationsfrom the driver. One or more cameras in the vehicle, for example, can beused to collect image data that assist in making this detection. In oneexample, the vehicle is configured in real-time to respond to thecollected user data (e.g., as analyzed in combination with data inputsregarding the user from the vehicle as currently being driven oroperated).

In yet other embodiments, the data collected is used to create a profilefor the user. In one example, the profile is stored at a cloud server(or other network computing device) and is associated with an account ofthe user. In another example, the user rents a vehicle which isconfigured for use by the user based on the profile stored at the cloudserver.

In one embodiment, analysis of electronic communications associated witha user (e.g., text communications sent by the user) includes naturallanguage processing (NLP). For example, text extracted from thecommunications can be analyzed to determine a personality type for usein configuring a vehicle. For example, the personality type can beaggressive or passive. In one embodiment, an NLP algorithm can be usedto match collected text to a personality type. In one example, NLPsemantic analysis is used to identify a sentiment or mood of the userbased on the communications.

In other embodiments, an ANN model trained to process input data basedon collected data of a user may be used in one of many devices, such asin vehicles having functions for autonomous driving and/or advanceddriver assistance system (ADAS), in connected home devices havingartificial intelligence (AI) functions, in industry 4.0 devices havingAI functions for automation and data exchange in manufacturing, etc.Thus, the techniques discussed herein in connection with configurationof vehicles can also be used with other intelligent devices, such asthose for network connected homes, robots, manufacturing, etc.

FIG. 1 illustrates a system to configure at least one action of avehicle based on collected data for a user of the vehicle, according toone embodiment. The system uses an Artificial Neural Network (ANN) modelaccording to one embodiment. The system of FIG. 1 includes a centralizedserver 101 in communication with a set of vehicles 111, . . . , 113 viaa communications network 102. One of the vehicles will be configured foruse by the user based on collected data associated with the user.

In one embodiment, data regarding a user of vehicle 111 is collectedfrom sensors located on a fixed structure of the user and/or a mobiledevice of the user. The collected data may optionally further includedata collected from sensors located in vehicle 111 and/or anothervehicle, each of which vehicles have been previously used by the user.The collected data is analyzed, for example, using a computer model suchas an artificial neural network (ANN) model.

In one embodiment, the collected data is provided as an input to the ANNmodel. For example, the ANN model can be executed on server 101 and/orvehicle 111. The vehicle 111 is controlled based on at least one outputfrom the ANN model. For example, this control includes performing one ormore actions based on the output. These actions can include, forexample, control of steering, braking, acceleration, and/or control ofother systems of vehicle 111 such as an infotainment system and/orcommunication device.

In one embodiment, vehicle 111 is an electric vehicle that is chargedusing a charging device of a fixed structure of the user such as a housein which the user stores the electric vehicle. A computing deviceassociated with the charging device can, for example, wirelesslytransmit data collected from the fixed structure and/or charging deviceto a communications interface of controller or other computing device ofvehicle 111. In some cases, data collected from the fixed structureincludes data collected by sensors that monitor physical activity of theuser inside the fixed structure.

In one embodiment, the server 101 includes a supervised training module117 to train, generate, and update ANN model 119 that includes neuronbiases 121, synaptic weights 123, and activation functions 125 ofneurons in a network used for processing collected data of a user and/orsensor data generated in the vehicles 111, . . . , 113. One or more ofthese vehicles can be, for example, previously driven by the user.

Once the ANN model 119 is designed, trained and implemented, e.g., forautonomous driving and/or advanced driver assistance system, the ANNmodel 119 can be deployed on one or more of vehicles 111, . . . , 113for real world usage by the user.

In various embodiments, the ANN model is trained using collected userdata. The training can be performed on a server and/or the vehicle.Configuration for an ANN model as used in a vehicle can be updated basedon the training. The training can be performed in some cases while thevehicle is being operated.

Typically, the vehicles 111, . . . , 113 have sensors, such as a visiblelight camera, an infrared camera, a LIDAR, a RADAR, a sonar, and/or aset of peripheral sensors. The sensors of the vehicles 111, . . . , 113generate sensor inputs for the ANN model 119 in autonomous drivingand/or advanced driver assistance system to generate operatinginstructions, such as steering, braking, accelerating, driving, alerts,emergency response, etc.

During the operations of the vehicles 111, . . . , 113 in theirrespective service environments, the vehicles 111, . . . , 113 encounteritems, such as events or objects, that are captured in the sensor data.The ANN model 119 is used by the vehicles 111, . . . , 113 to providethe identifications of the items to facilitate the generation ofcommands for the operations of the vehicles 111, . . . , 113, such asfor autonomous driving and/or for advanced driver assistance.

For example, a vehicle 111 may communicate, via a wireless connection115 to an access point (or base station) 105, with the server 101 tosubmit the sensor input to enrich the sensor data 103 as an additionaldataset for machine learning implemented using the supervised trainingmodule 117. The wireless connection 115 may be made via a wireless localarea network, a cellular communications network, and/or a communicationlink 107 to a satellite 109 or a communication balloon. In one example,user data collected from a vehicle can be similarly transmitted to theserver.

Optionally, the sensor input stored in the vehicle 111 may betransferred to another computer for uploading to the centralized server101. For example, the sensor input can be transferred to anothercomputer via a memory device, such as a Universal Serial Bus (USB)drive, and/or via a wired computer connection, a Bluetooth or WiFiconnection, a diagnosis tool, etc.

Periodically, the server 101 runs the supervised training module 117 toupdate the ANN model 119 based on updated collected data regarding theuser. The server 101 may use the sensor data 103 enhanced with thesensor inputs from the vehicle 111 based on prior operation by the userand/or from similar vehicles e.g., 113 that are operated in the samegeographical region or in geographical regions having similar trafficconditions to generate a customized version of the ANN model 119 for thevehicle 111.

Optionally, the server 101 uses the sensor data 103 enhanced with thesensor inputs from a general population of vehicles e.g., 111, 113 togenerate an updated version of the ANN model 119.

The updated ANN model 119 can be downloaded to the vehicle of the usere.g., 111 via the communications network 102, the access point (or basestation) 105, and communication links 115 and/or 117 as an over-the-airupdate of the firmware/software of the vehicles e.g., 111.

Optionally, the vehicle 111 has a self-learning capability. After anextended period on the road, the vehicle 111 may generate a new set ofsynaptic weights 123, neuron biases 121, activation functions 125,and/or neuron connectivity for the ANN model 119 installed in thevehicle 111 using the sensor inputs it collected and stored in thevehicle 111. As an example, the centralized server 101 may be operatedby a factory, a producer or maker of the vehicles 111, . . . , 113, or avendor of the autonomous driving and/or advanced driver assistancesystem for vehicles 111, . . . , 113.

FIG. 2 shows a vehicle that is configured using an Artificial NeuralNetwork (ANN) model trained using data that is collected for a user ofthe vehicle, according to one embodiment. The vehicle 111 of FIG. 2includes an infotainment system 149, a communication device 139, one ormore sensors 137, and a computer 131 that is connected to some controlsof the vehicle 111, such as a steering control 141 for the direction ofthe vehicle 111, a braking control 143 for stopping of the vehicle 111,an acceleration control 145 for the speed of the vehicle 111, etc.

The computer 131 of the vehicle 111 includes one or more processors 133,memory 135 storing firmware (or software) 127, the ANN model 119 (e.g.,as illustrated in FIG. 1), and other data 129.

The one or more sensors 137 may include a visible light camera, aninfrared camera, a LIDAR, RADAR, or sonar system, and/or peripheralsensors, which are configured to provide sensor input to the computer131. A module of the firmware (or software) 127 executed in theprocessor(s) 133 applies the sensor input to an ANN defined by the model119 to generate an output that identifies or classifies an event orobject captured in the sensor input, such as an image or video clip.

The identification or classification of the event or object generated bythe ANN model 119 can be used by an autonomous driving module of thefirmware (or software) 127, or an advanced driver assistance system, togenerate a response. The response may be a command to activate and/oradjust one of the vehicle controls 141, 143, and 145. In one embodiment,the generated response is an action performed by the vehicle where theaction has been configured based on analyzing collected data of theuser. Prior to generating the response, the vehicle the user isconfigured. In one embodiment, the configuration of the vehicle isperformed by updating firmware of vehicle 111 based on collected userdata. In one embodiment, the configuration of the vehicle includesupdating of the computer model stored in vehicle 111 e.g., ANN model119.

The server 101 stores the received sensor input as part of the sensordata 103 for the subsequent further training or updating of the ANNmodel 119 using the supervised training module 117.

When an updated version of the ANN model 119 is available in the server101, the vehicle 111 may use the communication device 139 to downloadthe updated ANN model 119 for installation in the memory 135 and/or forthe replacement of the previously installed ANN model 119.

In one example, the outputs of the ANN model 119 can be used to control(e.g., 141, 143, 145) the acceleration of a vehicle (e.g., 111), thespeed of the vehicle 111, and/or the direction of the vehicle 111,during autonomous driving or provision of advanced driver assistance.

Typically, when the ANN model is generated, at least a portion of thesynaptic weights 123 of some of the neurons in the network is updated.The update may also adjust some neuron biases 121 and/or change theactivation functions 125 of some neurons. In some instances, additionalneurons may be added in the network. In other instances, some neuronsmay be removed from the network.

In one example, data obtained from a sensor may be an image or videothat captures an event and/or an object using a camera that images usinglights visible to human eyes, or a camera that images using infraredlights, or a sonar, radar, or LIDAR system. In one embodiment, audiodata and/or image data obtained from at least one sensor of vehicle 111is part of the collected user data that was analyzed. In some instances,the ANN model is configured for a particular vehicle 111 based on thesensor and other collected data.

FIG. 3 shows a method to configure at least one action performed by avehicle based on collected data of a user, according to one embodiment.In block 601, data is collected regarding a user of a vehicle (e.g.,vehicle 111). In block 603, the collected data is analyzed (e.g., usingan ANN model). In block 605, at least one action performed by a vehicle(e.g., driving control or control of infotainment system 149) isconfigured based on analyzing the collected data.

In one embodiment, a method comprises collecting, by at least oneprocessor, data regarding a user of a vehicle; analyzing, by the atleast one processor, the collected data; and configuring, based onanalyzing the collected data, at least one action performed by thevehicle.

In one embodiment, the method further comprises performing voicerecognition using audio data associated with the user, wherein analyzingthe collected data comprises determining, at least in part based onperforming the voice recognition, a mood of the user.

In one embodiment, collecting the data comprises receiving first datafrom a computing device of a charger used to charge the vehicle.

In one embodiment, the charger is located within a fixed structure, thestructure comprises at least one sensor, and the first data is obtainedfrom the at least one sensor prior to configuring the at least oneaction.

In one embodiment, collecting the data comprises receiving data from anappliance of the user, wherein the appliance is located in a building ofthe user (e.g., a home of the user in which a vehicle of the user isstored and/or charged).

In one embodiment, the appliance is a security device, a refrigerator,an oven, a camera, or a voice recognition device. In one embodiment,collecting the data comprises collecting data from at least one sensorof a wearable computing device worn by the user.

In one embodiment, the method further comprises training a computermodel using at least one of supervised or unsupervised learning, whereinthe training is done using training data including at least a portion ofthe collected data.

In one embodiment, collecting the data comprises receiving image data oraudio data from at least one sensor.

In one embodiment, the collected data comprises image data, andanalyzing the collected data comprises performing facial recognition onthe image data to identify facial features for determining an emotionalstate of the user. In one embodiment, the collected data comprisesbiometric data of the user. In one embodiment, collecting the datacomprises receiving data from at least one of a motion detector, acamera, an accelerometer, or a microphone.

In one embodiment, the collected data comprises at least one of: datacollected by sensors that monitor at least one physical activity of theuser inside at least one fixed structure; data associated withelectronic communications of the user; data regarding times of day andcorresponding actions performed by the user; biometric data of the user;or data regarding input selections made by the user in a user interfaceof at least one of a vehicle or a computing device.

In one embodiment, analyzing the collected data comprises providing thecollected data as an input to a computer model, the method furthercomprising performing an action based on an output from the computermodel.

In one embodiment, the vehicle is an autonomous vehicle comprising acontroller, configuring the vehicle comprises updating firmware of thecontroller, and the updated firmware is stored in a storage device ofthe autonomous vehicle.

In one embodiment, a non-transitory computer storage medium storesinstructions which, when executed on a computing device, cause thecomputing device to perform a method comprising: collecting dataregarding a user of a vehicle, wherein the collected data comprises atleast one of data collected by sensors that monitor at least onephysical activity of a user, or data associated with electroniccommunications of the user on a mobile device; analyzing the collecteddata; and configuring, based on analyzing the collected data, at leastone action performed by the vehicle.

FIG. 4 shows a method to control a vehicle including performing anaction based on at least one output from a machine learning model,according to one embodiment. In block 611, a plurality of sensors areprovided on a fixed structure or a mobile device. For example, one ormore sensors can be mounted to a wall or other fixture inside the fixedstructure. In another example, one or more sensors are incorporated intoa mobile device of a user of a vehicle.

In block 613, data is collected from the sensors regarding a user ofvehicle. In block 615, the collected data is analyzed. This analysisincludes, for example, providing the data as an input to a machinelearning model.

In block 617, an action performed by a vehicle is controlled based on atleast one output from the machine learning model. For example, themachine learning model is ANN model 119.

In one embodiment, a system includes: a plurality of sensors located onat least one of a fixed structure or a mobile device; at least oneprocessor; and memory storing instructions configured to instruct the atleast one processor to: collect, from the plurality of sensors, dataregarding a user of a vehicle; analyze the collected data, wherein theanalyzing comprises providing the data as an input to a machine learningmodel; and control a vehicle, wherein the controlling comprisesperforming an action based on at least one output from the machinelearning model.

In one embodiment, the system further includes a communication interfaceconfigured to: wirelessly transmit the collected data to a computingdevice; and receive training data from the computing device; wherein aconfiguration of the machine learning model is updated using thetraining data.

In one embodiment, the training data includes data collected, prior tocontrolling the vehicle, from a charging device used to charge thevehicle.

In one embodiment, the machine learning model is trained using trainingdata, the training data comprising at least one of: data collected bysensors that monitor at least one physical activity of the user inside afixed structure; or data from electronic communications of the user.

FIG. 5 shows an autonomous vehicle 303 configured based on collecteduser data and controlled using a computer model, according to oneembodiment. At least a portion of the data for a user is collected fromthe vehicle itself. In one embodiment, a system controls a displaydevice 308 (or other device, system, or component) of an autonomousvehicle 303. For example, a controller 307 controls the display ofimages on one or more display devices 308.

The controller 307 may receive data collected by one or more sensors306. The sensors 306 may be, for example, mounted in the autonomousvehicle 303. The sensors 306 may include, for example, a camera, amicrophone, a motion detector, and/or a camera. The sensors 306 also mayinclude, for example, sensors incorporated in wearable devices worn bythe driver and/or passengers in the autonomous vehicle 303.

The sensors 306 may provide various types of data for collection by thecontroller 307. For example, the collected data may include image datafrom the camera and/or audio data from the microphone.

In one embodiment, the image data includes images of one or more facesof the driver and/or passengers (any one person of which can be a futureuser of vehicle 303 of FIG. 5 and/or a future user of vehicle 111 ofFIG. 1). In another embodiment, the collected data includes biometricdata for one or more persons in the autonomous vehicle 303. Thebiometric data may be provided, for example, by a wearable device. Inone embodiment, the display device 308 is an electroluminescent display(ELD).

In one embodiment, the controller 307 analyzes the collected data fromthe sensors 306 (and/or other collected data regarding a user). Theanalysis of the collected data includes providing some or all of thecollected data as one or more inputs to a computer model 312. Thecomputer model 312 can be, for example, an artificial neural networktrained by deep learning. In one example, the computer model is amachine learning model that is trained using training data 314. Thecomputer model 312 and/or the training data 314 can be stored, forexample, in memory 309.

In one embodiment, memory 309 stores a database 310, which may includedata collected by sensors 306 and/or data received by a communicationinterface 305 from computing device, such as, for example, a server 301(server 301 can be, for example, server 101 of FIG. 1 in someembodiments). In one example, this communication may be used towirelessly transmit collected data from the sensors 306 to the server301. The received data may include configuration, training, and otherdata used to configure control of the display devices 308 by controller307.

For example, the received data may include data collected from sensorsof autonomous vehicles other than autonomous vehicle 303. This data maybe included, for example, in training data 314 for training of thecomputer model 312. The received data may also be used to update aconfiguration of a machine learning model stored in memory 309 ascomputer model 312.

In FIG. 5, firmware 304 controls, for example, the operations of thecontroller 307 in controlling the display devices 308. The controller307 also can, for example, run the firmware 304 to perform operationsresponsive to communications from the server 301. The autonomous vehicle303 includes volatile Dynamic Random-Access Memory (DRAM) 311 for thestorage of run-time data and instructions used by the controller 307.

In one embodiment, memory 309 is implemented using variousmemory/storage technologies, such as NAND gate based flash memory,phase-change memory (PCM), magnetic memory (MRAM), resistiverandom-access memory, and 3D XPoint, such that the memory 309 isnon-volatile and can retain data stored therein without power for days,months, and/or years.

In one embodiment server 301 communicates with the communicationinterface 305 via a communication channel. In one embodiment, the server301 can be a computer having one or more Central Processing Units (CPUs)to which vehicles, such as the autonomous vehicle 303, may be connectedusing a computer network. For example, in some implementations, thecommunication channel between the server 301 and the communicationinterface 305 includes a computer network, such as a local area network,a wireless local area network, a cellular communications network, or abroadband high-speed always-connected wireless communication connection(e.g., a current or future generation of mobile network link).

In one embodiment, the controller 307 performs data intensive, in-memoryprocessing using data and/or instructions organized in memory 309 orotherwise organized in the autonomous vehicle 303. For example, thecontroller 307 can perform a real-time analysis of a set of datacollected and/or stored in the autonomous vehicle 303. In someembodiments, the set of data further includes collected user dataobtained from server 301.

At least some embodiments of the systems and methods disclosed hereincan be implemented using computer instructions executed by thecontroller 307, such as the firmware 304. In some instances, hardwarecircuits can be used to implement at least some of the functions of thefirmware 304. The firmware 304 can be initially stored in non-volatilestorage media, such as by using memory 309, or another non-volatiledevice, and loaded into the volatile DRAM 311 and/or the in-processorcache memory for execution by the controller 307. In one example, thefirmware 104 can be configured to use the techniques discussed hereinfor controlling display or other devices of a vehicle as configuredbased on collected user data.

FIG. 6 shows a vehicle 703 configured for a user based on collected dataof the user, according to one embodiment. The vehicle 703 includes acommunication interface 705 used to receive a configuration update,which is based on analysis of collected user data. For example, theupdate can be received from server 701 and/or client device 719.Communication amongst two or more of the vehicle 703, a server 701, anda client device 719 can be performed over a network 715 (e.g., awireless network). This communication is performed using communicationinterface 705.

In one embodiment, the server 701 controls the loading of user data(e.g., based on analysis of collected user data) of the new user intothe memory 709 of the vehicle. In one embodiment, data associated withusage of vehicle 703 is stored in a memory 721 of client device 719.

A controller 707 controls one or more operations of the vehicle 703. Forexample, controller 707 controls user data 714 stored in memory 709.Controller 707 also controls loading of updated configuration data intomemory 709 and/or other memory of the vehicle 703. Controller 707 alsocontrols display of information on display device(s) 708. Sensor(s) 706provide data regarding operation of the vehicle 703. At least a portionof this operational data can be communicated to the server 701 and/orthe client device 719.

Memory 709 can further include, for example, configuration data 712and/or database 710. Configuration data 712 can be, for example, dataassociated with operation of the vehicle 703 as provided by the server701. The configuration data 712 can be, for example, based on collecteduser data.

Database 710 can store, for example, configuration data for a userand/or data collected by sensors 706. Database 710 also can store, forexample, navigational maps and/or other data provided by the server 701.

In one embodiment, when a vehicle is being operated, data regardingactivity of vehicle 703 can be communicated to server 701. This activitymay include navigational and/or other operational aspects of the vehicle703 (e.g., as used by a user for which data is being collected).

As illustrated in FIG. 6, controller 707 also may control the display ofimages on one or more display devices 708. Display device 708 can be aliquid crystal display. The controller 707 may receive data collected byone or more sensors 706. The sensors 706 may be, for example, mounted inthe vehicle 703. The sensors 706 may include, for example, a camera, amicrophone, a motion detector, and/or a camera.

The sensors 706 may provide various types of data for collection by thecontroller 707. For example, the collected data may include image datafrom the camera and/or audio data from the microphone.

In one embodiment, the image data includes images of one or more facesof the driver and/or passengers. In another embodiment, the collecteddata includes biometric data for one or more persons in the vehicle 103.The biometric data may be provided, for example, by a wearable device.

In one embodiment, the controller 707 analyzes the collected data fromthe sensors 706. The analysis of the collected data includes providingsome or all of the collected data to server 701.

In one embodiment, server 701 analyzes collected data associated with auser of vehicle 703. Configuration data is generated based on theanalysis and then sent to vehicle 703.

In one embodiment, memory 709 stores database 710, which may includedata collected by sensors 706 and/or data received by communicationinterface 705 from a computing device, such as, for example, server 701.For example, this communication may be used to wirelessly transmitcollected data from the sensors 706 to the server 701. The data receivedby the vehicle may include configuration or other data used to configurecontrol of the display devices 708 by controller 707.

In FIG. 6, firmware 704 controls, for example, the operations of thecontroller 707. The controller 707 also can, for example, run thefirmware 704 to perform operations responsive to communications from theserver 701.

The vehicle 703 includes volatile Dynamic Random-Access Memory (DRAM)711 for the storage of run-time data and instructions used by thecontroller 707 to improve the computation performance of the controller707 and/or provide buffers for data transferred between the server 701and memory 709. DRAM 711 is volatile.

FIG. 7 is a block diagram of an autonomous vehicle including one or morevarious components and/or subsystems, each of which can be updated invarious embodiments to configure the vehicle (e.g., using a firmwareupdate) based on collected user data, as was described above. The systemillustrated in FIG. 7 may be installed entirely within a vehicle.

The system includes an autonomous vehicle subsystem 402. In theillustrated embodiment, autonomous vehicle subsystem 402 includes mapdatabase 402A, radar devices 402B, Lidar devices 402C, digital cameras402D, sonar devices 402E, GPS receivers 402F, and inertial measurementunits 402G. Each of the components of autonomous vehicle subsystem 402comprise standard components provided in most current autonomousvehicles. In one embodiment, map database 402A stores a plurality ofhigh-definition three-dimensional maps used for routing and navigation.Radar devices 402B, Lidar devices 402C, digital cameras 402D, sonardevices 402E, GPS receivers 402F, and inertial measurement units 402Gmay comprise various respective devices installed at various positionsthroughout the autonomous vehicle as known in the art. For example,these devices may be installed along the perimeter of an autonomousvehicle to provide location awareness, collision avoidance, and otherstandard autonomous vehicle functionality.

Vehicular subsystem 406 is additionally included within the system.Vehicular subsystem 406 includes various anti-lock braking systems 406A,engine control units 402B, and transmission control units 402C. Thesecomponents may be utilized to control the operation of the autonomousvehicle in response to the streaming data generated by autonomousvehicle subsystem 402A. The standard autonomous vehicle interactionsbetween autonomous vehicle subsystem 402 and vehicular subsystem 406 aregenerally known in the art and are not described in detail herein.

The processing side of the system includes one or more processors 410,short-term memory 412, an RF system 414, graphics processing units(GPUs) 416, long-term storage 418 and one or more interfaces 420.

The one or more processors 410 may comprise central processing units,FPGAs, or any range of processing devices needed to support theoperations of the autonomous vehicle. Memory 412 comprises DRAM or othersuitable volatile RAM for temporary storage of data required byprocessors 410. RF system 414 may comprise a cellular transceiver and/orsatellite transceiver. Long-term storage 418 may comprise one or morehigh-capacity solid-state drives (SSDs). In general, long-term storage418 may be utilized to store, for example, high-definition maps, routingdata, and any other data requiring permanent or semi-permanent storage.GPUs 416 may comprise one more high throughput GPU devices forprocessing data received from autonomous vehicle subsystem 402A.Finally, interfaces 420 may comprise various display units positionedwithin the autonomous vehicle (e.g., an in-dash screen).

The system additionally includes a reporting subsystem 404 whichperforms data collection (e.g., collection of data obtained from sensorsof the vehicle that is used to drive the vehicle). The reportingsubsystem 404 includes a sensor monitor 404A which is connected to bus408 and records sensor data transmitted on the bus 408 as well as anylog data transmitted on the bus. The reporting subsystem 404 mayadditionally include one or more endpoints to allow for systemcomponents to transmit log data directly to the reporting subsystem 404.

The reporting subsystem 404 additionally includes a packager 404B. Inone embodiment, packager 404B retrieves the data from the sensor monitor404A or endpoints and packages the raw data for transmission to acentral system (illustrated in FIG. 8). In some embodiments, packager404B may be configured to package data at periodic time intervals.Alternatively, or in conjunction with the foregoing, packager 404B maytransmit data in real-time and may compress data to facilitate real-timecommunications with a central system.

The reporting subsystem 404 additionally includes a batch processor404C. In one embodiment, the batch processor 404C is configured toperform any preprocessing on recorded data prior to transmittal. Forexample, batch processor 404C may perform compression operations on thedata prior to packaging by packager 404B. In another embodiment, batchprocessor 404C may be configured to filter the recorded data to removeextraneous data prior to packaging or transmittal. In anotherembodiment, batch processor 404C may be configured to perform datacleaning on the recorded data to conform the raw data to a formatsuitable for further processing by the central system.

Each of the devices is connected via a bus 408. In one embodiment, thebus 408 may comprise a controller area network (CAN) bus. In someembodiments, other bus types may be used (e.g., a FlexRay or MOST bus).Additionally, each subsystem may include one or more additional bussesto handle internal subsystem communications (e.g., LIN busses for lowerbandwidth communications).

FIG. 8 is a block diagram of a centralized autonomous vehicle operationssystem, according to various embodiments. As illustrated, the systemincludes a number of autonomous vehicles 502A-502E. In one embodiment,each autonomous vehicle may comprise an autonomous vehicle such as thatdepicted in FIG. 7. Each autonomous vehicle 502A-502E may communicatewith a central system 514 via a network 516. In one embodiment, network516 comprises a global network such as the Internet.

In one example, central system 514 is implemented using one or more ofservers 101, 301, and/or 701. In one example, one or more of autonomousvehicles 502A-502E are autonomous vehicle 703.

The system additionally includes a plurality of client devices 508A,508B. In the illustrated embodiment, client devices 508A, 508B maycomprise any personal computing device (e.g., a laptop, tablet, mobilephone, etc.). Client devices 508A, 508B may issue requests for data fromcentral system 514. In one embodiment, client devices 508A, 508Btransmit requests for data to support mobile applications or web pagedata, as described previously.

In one embodiment, central system 514 includes a plurality of servers504A. In one embodiment, servers 504A comprise a plurality of front endwebservers configured to serve responses to client device 508A, 508B.The servers 504A may additionally include one or more applicationservers configured to perform various operations to support one or morevehicles.

In one embodiment, central system 514 additionally includes a pluralityof models 504B. In one embodiment, models 504B may store one or moreneural networks for classifying autonomous vehicle objects. The models504B may additionally include models for predicting future events. Insome embodiments the models 504B may store a combination of neuralnetworks and other machine learning models.

Central system 514 additionally includes one or more databases 504C. Thedatabases 504C may include database record for vehicles 504D,personalities 504E, and raw data 504F. Raw data 504F may comprise anunstructured database for storing raw data received from sensors andlogs as discussed previously.

The present disclosure includes methods and apparatuses which performthese methods, including data processing systems which perform thesemethods, and computer readable media containing instructions which whenexecuted on data processing systems cause the systems to perform thesemethods.

Each of the server 101 and the computer 131 of a vehicle 111, . . . , or113 can be implemented as one or more data processing systems. A typicaldata processing system may include includes an inter-connect (e.g., busand system core logic), which interconnects a microprocessor(s) andmemory. The microprocessor is typically coupled to cache memory.

The inter-connect interconnects the microprocessor(s) and the memorytogether and also interconnects them to input/output (I/O) device(s) viaI/O controller(s). I/O devices may include a display device and/orperipheral devices, such as mice, keyboards, modems, network interfaces,printers, scanners, video cameras and other devices known in the art. Inone embodiment, when the data processing system is a server system, someof the I/O devices, such as printers, scanners, mice, and/or keyboards,are optional.

The inter-connect can include one or more buses connected to one anotherthrough various bridges, controllers and/or adapters. In one embodimentthe I/O controllers include a USB (Universal Serial Bus) adapter forcontrolling USB peripherals, and/or an IEEE-1394 bus adapter forcontrolling IEEE-1394 peripherals.

The memory may include one or more of: ROM (Read Only Memory), volatileRAM (Random Access Memory), and non-volatile memory, such as hard drive,flash memory, etc.

Volatile RAM is typically implemented as dynamic RAM (DRAM) whichrequires power continually in order to refresh or maintain the data inthe memory. Non-volatile memory is typically a magnetic hard drive, amagnetic optical drive, an optical drive (e.g., a DVD RAM), or othertype of memory system which maintains data even after power is removedfrom the system. The non-volatile memory may also be a random accessmemory.

The non-volatile memory can be a local device coupled directly to therest of the components in the data processing system. A non-volatilememory that is remote from the system, such as a network storage devicecoupled to the data processing system through a network interface suchas a modem or Ethernet interface, can also be used.

In the present disclosure, some functions and operations are describedas being performed by or caused by software code to simplifydescription. However, such expressions are also used to specify that thefunctions result from execution of the code/instructions by a processor,such as a microprocessor.

Alternatively, or in combination, the functions and operations asdescribed here can be implemented using special purpose circuitry, withor without software instructions, such as using Application-SpecificIntegrated Circuit (ASIC) or Field-Programmable Gate Array (FPGA).Embodiments can be implemented using hardwired circuitry withoutsoftware instructions, or in combination with software instructions.Thus, the techniques are limited neither to any specific combination ofhardware circuitry and software, nor to any particular source for theinstructions executed by the data processing system.

While one embodiment can be implemented in fully functioning computersand computer systems, various embodiments are capable of beingdistributed as a computing product in a variety of forms and are capableof being applied regardless of the particular type of machine orcomputer-readable media used to actually effect the distribution.

At least some aspects disclosed can be embodied, at least in part, insoftware. That is, the techniques may be carried out in a computersystem or other data processing system in response to its processor,such as a microprocessor, executing sequences of instructions containedin a memory, such as ROM, volatile RAM, non-volatile memory, cache or aremote storage device.

Routines executed to implement the embodiments may be implemented aspart of an operating system or a specific application, component,program, object, module or sequence of instructions referred to as“computer programs.” The computer programs typically include one or moreinstructions set at various times in various memory and storage devicesin a computer, and that, when read and executed by one or moreprocessors in a computer, cause the computer to perform operationsnecessary to execute elements involving the various aspects.

A machine readable medium can be used to store software and data whichwhen executed by a data processing system causes the system to performvarious methods. The executable software and data may be stored invarious places including for example ROM, volatile RAM, non-volatilememory and/or cache. Portions of this software and/or data may be storedin any one of these storage devices. Further, the data and instructionscan be obtained from centralized servers or peer to peer networks.Different portions of the data and instructions can be obtained fromdifferent centralized servers and/or peer to peer networks at differenttimes and in different communication sessions or in a same communicationsession. The data and instructions can be obtained in entirety prior tothe execution of the applications. Alternatively, portions of the dataand instructions can be obtained dynamically, just in time, when neededfor execution. Thus, it is not required that the data and instructionsbe on a machine readable medium in entirety at a particular instance oftime.

Examples of computer-readable media include but are not limited tonon-transitory, recordable and non-recordable type media such asvolatile and non-volatile memory devices, read only memory (ROM), randomaccess memory (RAM), flash memory devices, floppy and other removabledisks, magnetic disk storage media, optical storage media (e.g., CompactDisk Read-Only Memory (CD ROM), Digital Versatile Disks (DVDs), etc.),among others. The computer-readable media may store the instructions.

The instructions may also be embodied in digital and analogcommunication links for electrical, optical, acoustical or other formsof propagated signals, such as carrier waves, infrared signals, digitalsignals, etc. However, propagated signals, such as carrier waves,infrared signals, digital signals, etc. are not tangible machinereadable medium and are not configured to store instructions.

In general, a machine readable medium includes any mechanism thatprovides (i.e., stores and/or transmits) information in a formaccessible by a machine (e.g., a computer, network device, personaldigital assistant, manufacturing tool, any device with a set of one ormore processors, etc.).

In various embodiments, hardwired circuitry may be used in combinationwith software instructions to implement the techniques. Thus, thetechniques are neither limited to any specific combination of hardwarecircuitry and software nor to any particular source for the instructionsexecuted by the data processing system.

The above description and drawings are illustrative and are not to beconstrued as limiting. Numerous specific details are described toprovide a thorough understanding. However, in certain instances, wellknown or conventional details are not described in order to avoidobscuring the description. References to one or an embodiment in thepresent disclosure are not necessarily references to the sameembodiment; and, such references mean at least one.

In the foregoing specification, the disclosure has been described withreference to specific exemplary embodiments thereof. It will be evidentthat various modifications may be made thereto without departing fromthe broader spirit and scope as set forth in the following claims. Thespecification and drawings are, accordingly, to be regarded in anillustrative sense rather than a restrictive sense.

What is claimed is:
 1. A method comprising: collecting, by a vehicleusing at least one processor, data regarding a user of the vehicle,wherein collecting the data comprises collecting data by at least onesensor mounted on at least one wall of a fixed structure to monitor atleast one physical activity of the user, and collecting the data furthercomprises receiving first data from a computing device of a chargingdevice configured to charge the vehicle, the charging device locatedwithin the fixed structure; training, by the vehicle using a firstportion of the collected data regarding the user, a computer model;analyzing, by the at least one processor using the computer model, asecond portion of the collected data regarding the user; andconfiguring, based on at least one output from the computer model, adriving style of the vehicle, wherein configuring the driving stylecomprises updating firmware of a controller that controls accelerationof the vehicle.
 2. The method of claim 1, wherein collecting the dataregarding the user further comprises collecting audio data, andanalyzing the second portion comprises performing voice recognitionusing the audio data.
 3. The method of claim 1, wherein the firstportion is collected prior to configuring the driving style.
 4. Themethod of claim 1, wherein collecting the data further comprisesreceiving data from an appliance of the user, wherein the appliance islocated in the fixed structure.
 5. The method of claim 4, wherein theappliance is a security device, a refrigerator, an oven, a camera, or avoice recognition device.
 6. The method of claim 1, wherein collectingthe data further comprises collecting data from at least one sensor of awearable computing device worn by the user.
 7. The method of claim 1,wherein training the computer model is performed using at least one ofsupervised or unsupervised learning.
 8. The method of claim 1, whereincollecting the data further comprises receiving image data or audio datafrom at least one sensor of the vehicle.
 9. The method of claim 1,wherein the collected data regarding the user comprises image data, andanalyzing the second portion comprises performing facial recognition onthe image data to identify facial features for determining an emotionalstate of the user.
 10. The method of claim 1, wherein the secondportiondata comprises data from an accelerometer of the vehicle.
 11. Themethod of claim 1, wherein collecting the data regarding the userfurther comprises receiving data from at least one of a motion detector,a camera, an accelerometer, or a microphone.
 12. The method of claim 1,wherein the collected data regarding the user comprises data regardinginput selections made by the user in a user interface of the vehicle.13. The method of claim 1, wherein the vehicle is an autonomous vehicle,and the updated firmware is stored in a storage device of the autonomousvehicle.
 14. The method of claim 1, wherein the collected data regardingthe user further comprises data obtained from at a sensor of a wearablecomputing device worn by the user.
 15. A system comprising: a controllerof a vehicle, the controller configured by firmware to implement adriving style, the driving style including control of acceleration ofthe vehicle; at least one sensor located on a charging device configuredto charge the vehicle, the charging device located within a fixedstructure; at least one processor; and memory storing instructionsconfigured to instruct the at least one processor to: collect dataregarding a user of the vehicle, wherein the collected data comprisesdata collected to monitor at least one physical activity of the user byat least one sensor mounted on at least one wall of the fixed structure,and further comprises data collected by the at least one sensor of thecharging device; train, by the vehicle using a first portion of thecollected data regarding the user, a machine learning model; analyze asecond portion of the collected data regarding the user, wherein theanalyzing comprises providing the second portion as an input to themachine learning model; and update, based on at least one output fromthe machine learning model, the firmware of the controller to change thedriving style.
 16. The system of claim 15, wherein the second portioncomprises data from electronic communications of the user.
 17. Anon-transitory computer storage medium storing instructions which, whenexecuted on a computing device, cause the computing device to perform amethod comprising: collecting data regarding a user of a vehicle,wherein the collected data comprises data collected to monitor at leastone physical activity of the user by at least one sensor mounted on atleast one wall of a fixed structure, and further comprises datacollected by at least one sensor of a charging device that monitors atleast one physical activity of the user, wherein the charging device islocated within the fixed structure and configured to charge the vehicle;training, by the vehicle using a first portion of the collected dataregarding the user, a machine learning model; analyzing, using themachine learning model, a second portion of the collected data regardingthe user; and configuring, based on analyzing the second portion, adriving style of the vehicle, wherein configuring the driving stylecomprises updating firmware of a controller that controls accelerationof the vehicle.